Lockman SWIRE master catalogue

Checks and diagnostics

from herschelhelp_internal import git_version
print("This notebook was run with herschelhelp_internal version: \n{}".format(git_version()))

This notebook was run with herschelhelp_internal version: 
255270d (Fri Nov 24 10:35:51 2017 +0000)

%matplotlib inline
#%config InlineBackend.figure_format = 'svg'

import matplotlib.pyplot as plt
plt.rc('figure', figsize=(10, 6))
plt.style.use('ggplot')

import locale
locale.setlocale(locale.LC_ALL, 'en_GB')

import os
import time
import itertools

from astropy.coordinates import SkyCoord
from astropy.table import Table
from astropy import units as u
from astropy import visualization as vis
import numpy as np
from matplotlib_venn import venn3

from herschelhelp_internal.masterlist import (nb_compare_mags, nb_ccplots, nb_histograms, find_last_ml_suffix,
                                              quick_checks)

OUT_DIR = os.environ.get('OUT_DIR', "./data")
SUFFIX = find_last_ml_suffix()
#SUFFIX = "20170710"

master_catalogue_filename = "master_catalogue_lockman-swire_{}.fits".format(SUFFIX)
master_catalogue = Table.read("{}/{}".format(OUT_DIR, master_catalogue_filename))

print("Diagnostics done using: {}".format(master_catalogue_filename))

Diagnostics done using: master_catalogue_lockman-swire_20171129.fits

0 - Quick checks

quick_checks(master_catalogue)

The column ferr_ap_gpc1_g contains 123 zero or negative values!it's minimum is 0.0.
The column m_ap_gpc1_g contains 2 zero or negative values!it's minimum is -0.24098700284957886.
The column merr_ap_gpc1_g contains 123 zero or negative values!it's minimum is 0.0.
The column ferr_gpc1_g contains 262 zero or negative values!it's minimum is 0.0.
The column m_gpc1_g contains 3 zero or negative values!it's minimum is -0.9346719980239868.
The column merr_gpc1_g contains 262 zero or negative values!it's minimum is 0.0.
The column ferr_ap_gpc1_r contains 49 zero or negative values!it's minimum is 0.0.
The column merr_ap_gpc1_r contains 49 zero or negative values!it's minimum is 0.0.
The column ferr_gpc1_r contains 44 zero or negative values!it's minimum is 0.0.
The column merr_gpc1_r contains 44 zero or negative values!it's minimum is 0.0.
The column ferr_ap_gpc1_i contains 73 zero or negative values!it's minimum is 0.0.
The column merr_ap_gpc1_i contains 73 zero or negative values!it's minimum is 0.0.
The column ferr_gpc1_i contains 45 zero or negative values!it's minimum is 0.0.
The column merr_gpc1_i contains 45 zero or negative values!it's minimum is 0.0.
The column ferr_ap_gpc1_z contains 110 zero or negative values!it's minimum is 0.0.
The column merr_ap_gpc1_z contains 110 zero or negative values!it's minimum is 0.0.
The column ferr_gpc1_z contains 83 zero or negative values!it's minimum is 0.0.
The column merr_gpc1_z contains 83 zero or negative values!it's minimum is 0.0.
The column ferr_ap_gpc1_y contains 626 zero or negative values!it's minimum is 0.0.
The column m_ap_gpc1_y contains 1 zero or negative values!it's minimum is -0.782118022441864.
The column merr_ap_gpc1_y contains 626 zero or negative values!it's minimum is 0.0.
The column ferr_gpc1_y contains 234 zero or negative values!it's minimum is 0.0.
The column m_gpc1_y contains 1 zero or negative values!it's minimum is -0.6744740009307861.
The column merr_gpc1_y contains 234 zero or negative values!it's minimum is 0.0.
The column f_ap_irac_i1 contains 1885 zero or negative values!it's minimum is -9.76021327491179.
The column merr_ap_irac_i1 contains 1885 zero or negative values!it's minimum is -7642.055356778267.
The column f_irac_i1 contains 4025 zero or negative values!it's minimum is -564.766324700277.
The column merr_irac_i1 contains 4025 zero or negative values!it's minimum is -4156.577189059246.
The column f_ap_irac_i2 contains 11553 zero or negative values!it's minimum is -13.0778589205343.
The column merr_ap_irac_i2 contains 11553 zero or negative values!it's minimum is -11894.826131558493.
The column f_irac_i2 contains 5433 zero or negative values!it's minimum is -48.7146286757648.
The column merr_irac_i2 contains 5433 zero or negative values!it's minimum is -4842.104046291774.

I - Summary of wavelength domains

flag_obs = master_catalogue['flag_optnir_obs']
flag_det = master_catalogue['flag_optnir_det']

venn3(
    [
        np.sum(flag_obs == 4),
        np.sum(flag_obs == 2),
        np.sum(flag_obs == 6),
        np.sum(flag_obs == 1),
        np.sum(flag_obs == 5),
        np.sum(flag_obs == 3),
        np.sum(flag_obs == 7)
    ],
    set_labels=('Optical', 'near-IR', 'mid-IR'),
    subset_label_formatter=lambda x: "{}%".format(int(100*x/len(flag_obs)))
)
plt.title("Wavelength domain observations");

venn3(
    [
        np.sum(flag_det[flag_obs == 7] == 4),
        np.sum(flag_det[flag_obs == 7] == 2),
        np.sum(flag_det[flag_obs == 7] == 6),
        np.sum(flag_det[flag_obs == 7] == 1),
        np.sum(flag_det[flag_obs == 7] == 5),
        np.sum(flag_det[flag_obs == 7] == 3),
        np.sum(flag_det[flag_obs == 7] == 7)
    ],
    set_labels=('mid-IR', 'near-IR', 'Optical'),
    subset_label_formatter=lambda x: "{}%".format(int(100*x/np.sum(flag_det != 0)))
)
plt.title("Detection of the {} sources detected\n in any wavelength domains "
          "(among {} sources)".format(
              locale.format('%d', np.sum(flag_det != 0), grouping=True),
              locale.format('%d', len(flag_det), grouping=True)));

II - Comparing magnitudes in similar filters

The master list if composed of several catalogues containing magnitudes in similar filters on different instruments. We are comparing the magnitudes in these corresponding filters.

u_bands = ["WFC u",          "CFHT Megacam u"]
g_bands = ["WFC g", "RCS g", "CFHT Megacam g",  "GPC1 g"]
r_bands = ["WFC r", "RCS r", "CFHT Megacam r",  "GPC1 r"]
i_bands = ["WFC i", "RCS i",                    "GPC1 i"]
z_bands = ["WFC z", "RCS z", "CFHT Megacam z",  "GPC1 z"]
y_bands = [         "RCS y",                    "GPC1 y"]

II.a - Comparing depths

We compare the histograms of the total aperture magnitudes of similar bands.

for bands in [u_bands, g_bands, r_bands, i_bands, z_bands, y_bands]:
    colnames = ["m_{}".format(band.replace(" ", "_").lower()) for band in bands]
    nb_histograms(master_catalogue, colnames, bands)

II.b - Comparing magnitudes

We compare one to one each magnitude in similar bands.

for band_of_a_kind in [u_bands, g_bands, r_bands, i_bands, z_bands, y_bands]:
    for band1, band2 in itertools.combinations(band_of_a_kind, 2):
        
        basecol1, basecol2 = band1.replace(" ", "_").lower(), band2.replace(" ", "_").lower()
        
        # RCS has no aperture magnitudes
        if 'rcs' not in basecol1 and 'rcs' not in basecol2:
            col1, col2 = "m_ap_{}".format(basecol1), "m_ap_{}".format(basecol2)
            nb_compare_mags(master_catalogue[col1], master_catalogue[col2], 
                            labels=("{} (aperture)".format(band1), "{} (aperture)".format(band2)))
        
        col1, col2 = "m_{}".format(basecol1), "m_{}".format(basecol2)
        nb_compare_mags(master_catalogue[col1], master_catalogue[col2], 
                        labels=("{} (total)".format(band1), "{} (total)".format(band2)))

CFHT Megacam u (aperture) - WFC u (aperture):
- Median: -0.06
- Median Absolute Deviation: 0.25
- 1% percentile: -0.9491949844360351
- 99% percentile: 3.2663552665710673

CFHT Megacam u (total) - WFC u (total):
- Median: -0.27
- Median Absolute Deviation: 0.21
- 1% percentile: -1.6218692207336425
- 99% percentile: 2.0539311504363935

RCS g (total) - WFC g (total):
- Median: -0.08
- Median Absolute Deviation: 0.18
- 1% percentile: -0.9441658782958985
- 99% percentile: 1.1565348625183072

CFHT Megacam g (aperture) - WFC g (aperture):
- Median: 0.11
- Median Absolute Deviation: 0.14
- 1% percentile: -0.5078629302978516
- 99% percentile: 0.8724484062194824

CFHT Megacam g (total) - WFC g (total):
- Median: -0.04
- Median Absolute Deviation: 0.15
- 1% percentile: -0.8757493972778321
- 99% percentile: 1.4329431533813466

GPC1 g (aperture) - WFC g (aperture):
- Median: -0.14
- Median Absolute Deviation: 0.27
- 1% percentile: -2.033579864501953
- 99% percentile: 2.2427005767822266

GPC1 g (total) - WFC g (total):
- Median: -0.03
- Median Absolute Deviation: 0.24
- 1% percentile: -2.5378850173950194
- 99% percentile: 1.7712992477417002

CFHT Megacam g (total) - RCS g (total):
- Median: 0.05
- Median Absolute Deviation: 0.18
- 1% percentile: -1.0496853256225587
- 99% percentile: 1.2588200759887684

GPC1 g (total) - RCS g (total):
- Median: 0.04
- Median Absolute Deviation: 0.31
- 1% percentile: -3.0529003143310547
- 99% percentile: 1.945699691772461

GPC1 g (aperture) - CFHT Megacam g (aperture):
- Median: -0.33
- Median Absolute Deviation: 0.38
- 1% percentile: -2.9429248237609866
- 99% percentile: 2.1852089309692397

GPC1 g (total) - CFHT Megacam g (total):
- Median: 0.01
- Median Absolute Deviation: 0.27
- 1% percentile: -3.230147476196289
- 99% percentile: 1.8613366317748992

RCS r (total) - WFC r (total):
- Median: -0.13
- Median Absolute Deviation: 0.15
- 1% percentile: -1.01845121383667
- 99% percentile: 1.122300624847412

CFHT Megacam r (aperture) - WFC r (aperture):
- Median: 0.13
- Median Absolute Deviation: 0.15
- 1% percentile: -0.545387954711914
- 99% percentile: 1.148755531311045

CFHT Megacam r (total) - WFC r (total):
- Median: -0.08
- Median Absolute Deviation: 0.14
- 1% percentile: -0.9593395805358886
- 99% percentile: 1.6030628204345634

GPC1 r (aperture) - WFC r (aperture):
- Median: -0.10
- Median Absolute Deviation: 0.15
- 1% percentile: -1.2963315582275392
- 99% percentile: 1.1390761566162118

GPC1 r (total) - WFC r (total):
- Median: -0.01
- Median Absolute Deviation: 0.14
- 1% percentile: -1.7165836334228515
- 99% percentile: 0.9766998291015625

CFHT Megacam r (total) - RCS r (total):
- Median: 0.07
- Median Absolute Deviation: 0.17
- 1% percentile: -1.103610725402832
- 99% percentile: 1.3413280487060542

GPC1 r (total) - RCS r (total):
- Median: 0.13
- Median Absolute Deviation: 0.14
- 1% percentile: -1.8510711669921875
- 99% percentile: 1.0487995147705078

GPC1 r (aperture) - CFHT Megacam r (aperture):
- Median: -0.26
- Median Absolute Deviation: 0.25
- 1% percentile: -2.3193286895751952
- 99% percentile: 1.125625000000003

GPC1 r (total) - CFHT Megacam r (total):
- Median: 0.07
- Median Absolute Deviation: 0.14
- 1% percentile: -2.370811748504639
- 99% percentile: 1.0244120407104438

RCS i (total) - WFC i (total):
- Median: -0.08
- Median Absolute Deviation: 0.20
- 1% percentile: -0.9779530334472656
- 99% percentile: 1.4092513275146472

GPC1 i (aperture) - WFC i (aperture):
- Median: -0.03
- Median Absolute Deviation: 0.14
- 1% percentile: -1.025598430633545
- 99% percentile: 0.9575976371765118

GPC1 i (total) - WFC i (total):
- Median: 0.06
- Median Absolute Deviation: 0.15
- 1% percentile: -1.102666082382202
- 99% percentile: 0.9996103668212877

GPC1 i (total) - RCS i (total):
- Median: 0.17
- Median Absolute Deviation: 0.12
- 1% percentile: -1.6147754669189454
- 99% percentile: 0.8306443786621096

RCS z (total) - WFC z (total):
- Median: -0.05
- Median Absolute Deviation: 0.24
- 1% percentile: -0.999578857421875
- 99% percentile: 1.782779693603516

CFHT Megacam z (aperture) - WFC z (aperture):
- Median: 0.14
- Median Absolute Deviation: 0.17
- 1% percentile: -0.6397380828857422
- 99% percentile: 3.043675384521484

CFHT Megacam z (total) - WFC z (total):
- Median: -0.04
- Median Absolute Deviation: 0.22
- 1% percentile: -0.9713739395141602
- 99% percentile: 2.110036201477051

GPC1 z (aperture) - WFC z (aperture):
- Median: 0.01
- Median Absolute Deviation: 0.15
- 1% percentile: -0.9230419921874999
- 99% percentile: 1.1071548843383785

GPC1 z (total) - WFC z (total):
- Median: 0.09
- Median Absolute Deviation: 0.19
- 1% percentile: -0.9829817771911621
- 99% percentile: 1.4533742713928244

CFHT Megacam z (total) - RCS z (total):
- Median: 0.05
- Median Absolute Deviation: 0.24
- 1% percentile: -1.255912399291992
- 99% percentile: 2.259424972534189

GPC1 z (total) - RCS z (total):
- Median: 0.15
- Median Absolute Deviation: 0.18
- 1% percentile: -1.9935058212280274
- 99% percentile: 1.3433990478515625

GPC1 z (aperture) - CFHT Megacam z (aperture):
- Median: -0.22
- Median Absolute Deviation: 0.26
- 1% percentile: -2.777841310501098
- 99% percentile: 1.4052535629272451

GPC1 z (total) - CFHT Megacam z (total):
- Median: 0.12
- Median Absolute Deviation: 0.18
- 1% percentile: -2.3167495727539062
- 99% percentile: 1.362018585205071

GPC1 y (total) - RCS y (total):
- Median: -0.57
- Median Absolute Deviation: 0.41
- 1% percentile: -3.220444412231445
- 99% percentile: 1.1729692840576167

III - Comparing magnitudes to reference bands

Cross-match the master list to SDSS and 2MASS to compare its magnitudes to SDSS and 2MASS ones.

master_catalogue_coords = SkyCoord(master_catalogue['ra'], master_catalogue['dec'])

III.a - Comparing u, g, r, i, and z bands to SDSS

The catalogue is cross-matched to SDSS-DR13 withing 0.2 arcsecond.

We compare the u, g, r, i, and z magnitudes to those from SDSS using fiberMag for the aperture magnitude and petroMag for the total magnitude.

sdss = Table.read("../../dmu0/dmu0_SDSS-DR13/data/SDSS-DR13_Lockman-SWIRE.fits")
sdss_coords = SkyCoord(sdss['ra'] * u.deg, sdss['dec'] * u.deg)

idx, d2d, _ = sdss_coords.match_to_catalog_sky(master_catalogue_coords)
mask = (d2d < 0.2 * u.arcsec)

sdss = sdss[mask]
ml_sdss_idx = idx[mask]

for band_of_a_kind in [u_bands, g_bands, r_bands, i_bands, z_bands]:
    for band in band_of_a_kind:
        
        sdss_mag_ap_colname = "fiberMag_{}".format(band[-1])
        sdss_mag_tot_colname = "petroMag_{}".format(band[-1])
        master_cat_mag_ap_colname = "m_ap_{}".format(band.replace(" ", "_").lower())
        master_cat_mag_tot_colname = "m_{}".format(band.replace(" ", "_").lower())
        
        # Set SDSS magnitudes to NaN when the magnitude  is negative as SDSS uses large
        # negative numbers for missing magnitudes
        sdss[sdss_mag_ap_colname][sdss[sdss_mag_ap_colname] < 0.] = np.nan
        sdss[sdss_mag_tot_colname][sdss[sdss_mag_tot_colname] < 0.] = np.nan
  
        # RCS has no aperture magnitudes
        if 'rcs' not in master_cat_mag_ap_colname:
            sdss_mag_ap = sdss[sdss_mag_ap_colname]
            master_cat_mag_ap = master_catalogue[master_cat_mag_ap_colname][ml_sdss_idx]
    
            nb_compare_mags(sdss_mag_ap, master_cat_mag_ap,
                            labels=("SDSS {} (fiberMag)".format(band[-1]), "{} (aperture)".format(band)))
    
        sdss_mag_tot = sdss[sdss_mag_tot_colname]
        master_cat_mag_tot = master_catalogue[master_cat_mag_tot_colname][ml_sdss_idx]
        
        nb_compare_mags(sdss_mag_tot, master_cat_mag_tot,
                        labels=("SDSS {} (petroMag)".format(band[-1]), "{} (total)".format(band)))

WFC u (aperture) - SDSS u (fiberMag):
- Median: -0.31
- Median Absolute Deviation: 0.18
- 1% percentile: -1.5710824012756348
- 99% percentile: 0.7945964813232422

WFC u (total) - SDSS u (petroMag):
- Median: 0.07
- Median Absolute Deviation: 0.36
- 1% percentile: -3.809451866149902
- 99% percentile: 1.9933301925659166

CFHT Megacam u (aperture) - SDSS u (fiberMag):
- Median: 0.01
- Median Absolute Deviation: 0.52
- 1% percentile: -1.7239515686035158
- 99% percentile: 3.037174682617188

CFHT Megacam u (total) - SDSS u (petroMag):
- Median: 0.09
- Median Absolute Deviation: 0.83
- 1% percentile: -4.138626194000244
- 99% percentile: 4.583783168792724

WFC g (aperture) - SDSS g (fiberMag):
- Median: -0.45
- Median Absolute Deviation: 0.12
- 1% percentile: -1.153176498413086
- 99% percentile: 0.4261890411376952

WFC g (total) - SDSS g (petroMag):
- Median: -0.08
- Median Absolute Deviation: 0.23
- 1% percentile: -3.3376033782958983
- 99% percentile: 1.32151603698731

RCS g (total) - SDSS g (petroMag):
- Median: -0.20
- Median Absolute Deviation: 0.29
- 1% percentile: -3.748886833190918
- 99% percentile: 1.6347195434570319

CFHT Megacam g (aperture) - SDSS g (fiberMag):
- Median: -0.29
- Median Absolute Deviation: 0.18
- 1% percentile: -1.0998111724853517
- 99% percentile: 1.262142181396477

CFHT Megacam g (total) - SDSS g (petroMag):
- Median: -0.14
- Median Absolute Deviation: 0.25
- 1% percentile: -3.455993041992188
- 99% percentile: 2.2379463577270533

GPC1 g (aperture) - SDSS g (fiberMag):
- Median: -0.56
- Median Absolute Deviation: 0.26
- 1% percentile: -2.496891307830811
- 99% percentile: 1.7368188095092725

GPC1 g (total) - SDSS g (petroMag):
- Median: -0.10
- Median Absolute Deviation: 0.27
- 1% percentile: -4.111917877197266
- 99% percentile: 1.7278203964233376

WFC r (aperture) - SDSS r (fiberMag):
- Median: -0.35
- Median Absolute Deviation: 0.07
- 1% percentile: -0.9245659255981445
- 99% percentile: 0.3214846229553226

WFC r (total) - SDSS r (petroMag):
- Median: 0.04
- Median Absolute Deviation: 0.15
- 1% percentile: -2.676791229248047
- 99% percentile: 1.1676996612548833

/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/ipykernel/__main__.py:11: RuntimeWarning: invalid value encountered in less

RCS r (total) - SDSS r (petroMag):
- Median: -0.11
- Median Absolute Deviation: 0.16
- 1% percentile: -2.9832832145690915
- 99% percentile: 1.1344312286376967

CFHT Megacam r (aperture) - SDSS r (fiberMag):
- Median: -0.21
- Median Absolute Deviation: 0.14
- 1% percentile: -0.8575413894653321
- 99% percentile: 2.322139625549326

CFHT Megacam r (total) - SDSS r (petroMag):
- Median: -0.05
- Median Absolute Deviation: 0.15
- 1% percentile: -2.8071770668029785
- 99% percentile: 2.129177074432381

GPC1 r (aperture) - SDSS r (fiberMag):
- Median: -0.43
- Median Absolute Deviation: 0.14
- 1% percentile: -1.5779515075683594
- 99% percentile: 0.6068578338623037

GPC1 r (total) - SDSS r (petroMag):
- Median: 0.04
- Median Absolute Deviation: 0.14
- 1% percentile: -2.5192892456054685
- 99% percentile: 1.1319837570190423

WFC i (aperture) - SDSS i (fiberMag):
- Median: -0.41
- Median Absolute Deviation: 0.08
- 1% percentile: -0.9962218856811523
- 99% percentile: 0.35475969314575084

WFC i (total) - SDSS i (petroMag):
- Median: -0.02
- Median Absolute Deviation: 0.18
- 1% percentile: -3.216203365325928
- 99% percentile: 1.130485744476317

RCS i (total) - SDSS i (petroMag):
- Median: -0.13
- Median Absolute Deviation: 0.17
- 1% percentile: -3.6602382850646973
- 99% percentile: 1.091551609039307

GPC1 i (aperture) - SDSS i (fiberMag):
- Median: -0.41
- Median Absolute Deviation: 0.12
- 1% percentile: -1.1483927154541016
- 99% percentile: 0.3729707336425758

GPC1 i (total) - SDSS i (petroMag):
- Median: 0.05
- Median Absolute Deviation: 0.13
- 1% percentile: -2.7237721443176266
- 99% percentile: 0.9462432289123528

WFC z (aperture) - SDSS z (fiberMag):
- Median: -0.35
- Median Absolute Deviation: 0.12
- 1% percentile: -1.2592236328125
- 99% percentile: 0.6335509490966799

WFC z (total) - SDSS z (petroMag):
- Median: 0.00
- Median Absolute Deviation: 0.30
- 1% percentile: -4.017242908477783
- 99% percentile: 1.5223383903503418

RCS z (total) - SDSS z (petroMag):
- Median: -0.04
- Median Absolute Deviation: 0.40
- 1% percentile: -4.116477966308594
- 99% percentile: 2.0090103149414062

CFHT Megacam z (aperture) - SDSS z (fiberMag):
- Median: -0.14
- Median Absolute Deviation: 0.20
- 1% percentile: -1.1127138900756837
- 99% percentile: 2.7650779533386163

CFHT Megacam z (total) - SDSS z (petroMag):
- Median: -0.01
- Median Absolute Deviation: 0.35
- 1% percentile: -3.9249173927307126
- 99% percentile: 2.2055882263183637

GPC1 z (aperture) - SDSS z (fiberMag):
- Median: -0.32
- Median Absolute Deviation: 0.17
- 1% percentile: -1.417550163269043
- 99% percentile: 1.0491362571716336

GPC1 z (total) - SDSS z (petroMag):
- Median: 0.11
- Median Absolute Deviation: 0.29
- 1% percentile: -3.821057033538818
- 99% percentile: 1.7081263542175282

III.b - Comparing J and K bands to 2MASS

The catalogue is cross-matched to 2MASS-PSC withing 0.2 arcsecond. We compare the UKIDSS total J and K magnitudes to those from 2MASS.

The 2MASS magnitudes are “Vega-like” and we have to convert them to AB magnitudes using the zero points provided on this page:

Band	Fν - 0 mag (Jy)
J	1594
H	1024
Ks	666.7

In addition, UKIDSS uses a K band whereas 2MASS uses a Ks (“short”) band, this page give a correction to convert the K band in a Ks band with the formula:

$$K_{s(2MASS)} = K_{UKIRT} + 0.003 + 0.004 * (J−K)_{UKIRT}$$

# The AB zero point is 3631 Jy
j_2mass_to_ab = 2.5 * np.log10(3631/1595)
k_2mass_to_ab = 2.5 * np.log10(3631/666.7)

twomass = Table.read("../../dmu0/dmu0_2MASS-point-sources/data/2MASS-PSC_Lockman-SWIRE.fits")
twomass_coords = SkyCoord(twomass['raj2000'], twomass['dej2000'])

idx, d2d, _ = twomass_coords.match_to_catalog_sky(master_catalogue_coords)
mask = (d2d < 0.2 * u.arcsec)

twomass = twomass[mask]
ml_twomass_idx = idx[mask]

nb_compare_mags(twomass['jmag'] + j_2mass_to_ab, master_catalogue['m_wfcam_j'][ml_twomass_idx],
                labels=("2MASS J", "WFCAM J (total)"))

WFCAM J (total) - 2MASS J:
- Median: 0.02
- Median Absolute Deviation: 0.05
- 1% percentile: -0.9130657836820437
- 99% percentile: 0.575751805124098

wfcam_ks_like = master_catalogue['m_wfcam_k'] + 0.003 + 0.004 * (
    master_catalogue['m_wfcam_j'] - master_catalogue['m_wfcam_k'])
nb_compare_mags(twomass['kmag'] + k_2mass_to_ab, wfcam_ks_like[ml_twomass_idx],
                labels=("2MASS Ks", "WFCAM Ks-like (total)"))

WFCAM Ks-like (total) - 2MASS Ks:
- Median: 0.07
- Median Absolute Deviation: 0.08
- 1% percentile: -0.8240238957842398
- 99% percentile: 0.9100533730451055

Keeping only sources with good signal to noise ratio

From here, we are only comparing sources with a signal to noise ratio above 3, i.e. roughly we a magnitude error below 0.3.

To make it easier, we are setting to NaN in the catalogue the magnitudes associated with an error above 0.3 so we can't use these magnitudes after the next cell.

for error_column in [_ for _ in master_catalogue.colnames if _.startswith('merr_')]:
    column = error_column.replace("merr", "m")
    keep_mask = np.isfinite(master_catalogue[error_column])
    keep_mask[keep_mask] &= master_catalogue[keep_mask][error_column] <= 0.3
    master_catalogue[column][~keep_mask] = np.nan

IV - Comparing aperture magnitudes to total ones.

nb_ccplots(
    master_catalogue['m_cfht_megacam_r'],
    master_catalogue['m_ap_cfht_megacam_r'] - master_catalogue['m_cfht_megacam_r'],
    "r total magnitude (CFHT)", "r aperture mag - total mag (CFHT)",
    master_catalogue["stellarity"],
    invert_x=True
)

Number of source used: 2456857 / 4366298 (56.27%)

V - Color-color and magnitude-color plots

nb_ccplots(
    master_catalogue['m_wfc_g'] - master_catalogue['m_wfc_i'],
    master_catalogue['m_wfcam_j'] - master_catalogue['m_wfcam_k'],
    "g - i (WFC)", "J - K (UKIDSS)",
    master_catalogue["stellarity"]
)

Number of source used: 85805 / 4366298 (1.97%)

nb_ccplots(
    master_catalogue['m_wfc_i'] - master_catalogue['m_irac_i1'],
    master_catalogue['m_wfc_g'] - master_catalogue['m_wfc_i'],
    "WFC i - IRAC1", "g - i (WFC)",
    master_catalogue["stellarity"]
)

Number of source used: 129379 / 4366298 (2.96%)

nb_ccplots(
    master_catalogue['m_cfht_megacam_u'] - master_catalogue['m_cfht_megacam_g'],
    master_catalogue['m_cfht_megacam_g'] - master_catalogue['m_cfht_megacam_r'],
    "u - g (CFHT)", "g - r (CFHT)",
    master_catalogue["stellarity"]
)

Number of source used: 2070789 / 4366298 (47.43%)

nb_ccplots(
    master_catalogue['m_wfcam_j'] - master_catalogue['m_wfcam_k'],
    master_catalogue['m_wfc_g'] - master_catalogue['m_wfcam_j'],
    "J - K (WFCAM)", "g - J (WFC, WFCAM)",
    master_catalogue["stellarity"]
)

Number of source used: 108739 / 4366298 (2.49%)

nb_ccplots(
    master_catalogue['m_wfc_i'] - master_catalogue['m_wfc_z'],
    master_catalogue['m_wfc_z'] - master_catalogue['m_wfcam_j'],
    "i - z (WFC)", "z - J (WFC, WFCAM)",
    master_catalogue["stellarity"]
)

Number of source used: 11038 / 4366298 (0.25%)

nb_ccplots(
    master_catalogue['m_irac_i3'] - master_catalogue['m_irac_i4'],
    master_catalogue['m_irac_i1'] - master_catalogue['m_irac_i2'],
    "IRAC3 - IRAC4", "IRAC1 - IRAC2",
    master_catalogue["stellarity"]
)

Number of source used: 38805 / 4366298 (0.89%)